We were tasked with two parts. First of all we had to connect a sensor and combine it with a processing visualisation, the looks of which we would be able to manipulate

The second involved a custom music track that via an analyzer plugin in Max 7, we would be able to manipulate the visualisations in a more distinct way, by the use of analysed features the music track came with




The first challenge was to connect the Processing sketch with an Arduino board. One method everyone used was firmata, however, since I have allergy to firmata (it does not work with many libraries) out of the box, I decided to reuse the code I developed a week ago, which utilised a fully working serial communication between an Arduino, Processing sketch and Max 7 via OCP/UDP protocols. Once I recreated my steps, I was able to have a fully functioning visualisation manipulator in motion, based on the distance the ultrasonic sensor picked up. The effect was subliminal, but relatively surprising at the same time

The largest challenge came once I had to involve an analyzer plugin, one that would be able to analyze any song possible, break it down to specific features with which I would be able to control visualisations more elaborately. Unfortunately, I never managed to get the plugin working. I looked for solutions on the web, however, despite having broached quite a few discussion boards, I was never able to get the plugin working nor find a suitable alternative, sadly



I was able to introduce a serial communication and an OSC/UDP protocol between Max, Processing and Arduino and successfully manipulate a processing sketch by picking up information from a variety of Arduino modules

I have also explored a variety of filters and ways of breaking down songs, along with rediscovering the existance of the very same filters available on PC, although sadly not working at this time


Hm.


I was not able to analyze data from any song due to a plugin compatibility-related issue and spend most of the time in class trying to figure out how to get it properly installed and working on my PC, as the Mac I was stationed at did not work at all, unfortunately.

Despite having found an equivalent that should have worked on PCs, it eventually didn't. Most likely due to the requirements targetting a Windows 7 PC, as some of the files that had to be altered were derived from previous setups, not suitable for my current system


//Processing Code
import processing.serial.*;
import oscP5.*;
import netP5.*;


OscP5 oscP5;

/* a NetAddress contains the ip address and port number of a remote location in the network. */
NetAddress myBroadcastLocation;

float sendOSCMouseX, sendOSCMouseY;

Serial myPort1;  // Create object from Serial class
Serial myPort2;
Serial myPort3;
String val1;     // Data received from the serial port
String val2;
String val3;

void setup()
{
  size(400,400);
  // I know that the first port in the serial list on my mac
  // is Serial.list()[0].
  // On Windows machines, this generally opens COM1.
  // Open whatever port is the one you're using.
  String portName = Serial.list()[0]; //change the 0 to a 1 or 2 etc. to match your port
  myPort1 = new Serial(this, portName, 115200);
  
  frameRate(25);

  /* create a new instance of oscP5. 
   * 12000 is the port number you are listening for incoming osc messages.
   */
  oscP5 = new OscP5(this, 12000);

  /* create a new NetAddress. a NetAddress is used when sending osc messages
   * with the oscP5.send method.
   */

  /* the address of the osc broadcast server */
  myBroadcastLocation = new NetAddress("127.0.0.1", 32000);
}

void draw()
{
  
    background(0);
  fill(45, 124, 26);
  noStroke();
  sendOSCMouseX = mouseX;
  sendOSCMouseY = mouseY;
  ellipse(sendOSCMouseX, sendOSCMouseY, 50, 50);
  
  OscMessage myOscMessage = new OscMessage("/test");
  myOscMessage.add(sendOSCMouseX/100);
  myOscMessage.add(sendOSCMouseY/100);
  String val;
  val = myPort1.readStringUntil('\n'); 
  val1 = trim(val);
  if (val==null || val=="1"){
  }
  else{
    float val2 = parseFloat(val1);
println(parseInt(val1));
  myOscMessage.add(val2/100);
  
   oscP5.send(myOscMessage, myBroadcastLocation);
  }
}




void mousePressed() {
  /* create a new OscMessage with an address pattern, in this case /test. */
  
  /* add a value (an integer) to the OscMessage */
  
  /* send the OscMessage to a remote location specified in myNetAddress */
 
}


void keyPressed() {
  OscMessage m;
  switch(key) {
    case('c'):
    /* connect to the broadcaster */
    m = new OscMessage("/server/connect", new Object[0]);
    oscP5.flush(m, myBroadcastLocation);  
    break;
    case('d'):
    /* disconnect from the broadcaster */
    m = new OscMessage("/server/disconnect", new Object[0]);
    oscP5.flush(m, myBroadcastLocation);  
    break;
  }
}


/* incoming osc message are forwarded to the oscEvent method. */
void oscEvent(OscMessage theOscMessage) {
  /* get and print the address pattern and the typetag of the received OscMessage */
  println("### received an osc message with addrpattern "+theOscMessage.addrPattern()+" and typetag "+theOscMessage.typetag());
  theOscMessage.print();
}



I would not have done anything differently actually, except perhaps, if needed, bought a Macintosh computer and worked on it, as apparently analyzer did eventually launch on this particular brand. That having said, I probably would have dedicated more time into finding an equivalent music-analyzing plugin or perhaps a legitimate solution. It might be the case that the plugin will not work in windows-based computers with the newest Max installed, plugins can be quite difficult to work with when they're not updated for quite some time.



It certainly does in a few dimensions. One of our ideas was to include a projection mapping placed underneath the umbrella. In fact, any projection mapping being showcased could potentially be manipulated with data included. Furthermore, we are involving sounds in our project, ie. nature sounds, which could potentially be analyzed via Max with sensors pitching another dimensions affecting specific visualisations.

To be honest, the potential applications are endless, however we need to consider the complexity of our project, as we were frequently reminded to not make it too complicated. Moreover, we are concentrating on addressing real user-needs bound with the area, in such case we will focus on tweaking these. That having said, we will keep in mind the affordances provided by the implication of Max and OSC/UDP protocols to send data back and forth. Despite that it may end up being a relatively tiring process, the gains are substantial, as it could even work remotely over a server if needed. With sensors properly installed, a simple IOT system could be established, although the question remains of how useful would it be for our project?



Well, the exercise could involve more sensors to begin with. Secondly, the visualisations could be more straightforward as to suggest direct user feedback, which it did not quite suggest throughout the entire session. Perhaps integrating the system with OpenCV could emerge with quite a few particularly interesting use cases, although at this point we would be working with such a multitude of dimensions that the effect a user could impose would eventually not be quite visible as we would have intended as.

It may be a different story if we involved more precise or a cluster of sensors. It all really comes down to the sensors really, as the communication could be even established via regular cables. Nevertheless it is just interesting to see how easy/intricate a system can be and through the lenses of our projects and the Lab's tutorials consider how other, potentially imaginary and visionary projects such as Amazon's shop in which you don't have to scan any products, you just pick them up and leave with Amazon automatically charging for the products, can conceptually be implemented. Broadening the horizons and showcasing more possibilities along with functions and the technical implementation is useful to comprehend phenomena taking place around us, as well as being able to contribute to a variety of potential projects.



All in all, I feel the tutorials in which we were taught how to communicate between devices were substantial. On one hand we need to know which sensors to use in order to evaluate and directly relate to a problem. And then see how we could solve it via communication and technical implementation means and provide a positive impact on the users.

Subliminal references:

Links to an external site.

Links to an external site.